Fine-Grained Micro-Tasks for MapReduce Skew-Handling

Recent work on MapReduce has considered the problems of skew, where a job’s tasks exhibit large variance in size and processing cost, and stragglers, tasks that run slowly due to conditions on particular nodes. In this paper, we discuss an extremely simple approach to mitigating skew and stragglers: break the workload into many small tasks that are dynamically scheduled at runtime. This approach is only effective in systems with high-throughput, low-latency task schedulers and efficient data materialization, so we propose techniques for scaling these components. To demonstrate the efficacy of this technique, we compare micro-tasks to other skew handling techniques using the Spark cluster computing framework.